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SOFTWARE  SUPPORT  RESOURCES  EVALUATION  GUIDE 


The  purpose  of  this  pamphlet  is  to  provide  Air  Force  Operational  Test  and  Evaluation  Center  (AFOTEC)  personnel 
information  needed  to  evaluate  software  support  resources  (SSR)  as  they  influence  overall  software  supportability. 
This  pamphlet  describes  how  to  plan,  conduct  and  report  a  software  support  resources  evaluation,  and  contains  a 
standardized  questionnaire  that  provides  a  framework  for  obtaining  test  team  evaluator  ratings  of  the  adequacy  of 
planned  or  existing  software  support  resources. 

This  volume  is  number  five  in  a  series  of  software  operational  test  and  evaluation  guides  prepared  by  the  Software 
Analysis  Team  at  Headquarters  (HQ)  AFOTEC.  Local  reproduction  of  all  volumes  in  this  series  is  authorized.  This 
volume  is  an  evolutionary  document  that  will  be  updated  periodically.  Comments  should  be  directed  to  the  office  of 
primary  responsibility  (OPR). 

SUMMARY  OF  CHANGES 

AFOTEC  Pamphlet  (AFOTECPAM)  99-102  replaces  AFOTEC  Pamphlet  800-2,  all  volumes.  This  volume  has  been 
completely  rewritten. 
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Chapter  1 
INTRODUCTION 


1.1  General.  This  pamphlet  describes  how  to  plan, 
conduct,  and  report  a  software  support  resources 
evaluation  in  support  of  AFOTEC-conducted 
operational  test  and  evaluation.  Contained  in  this 
publication  is  a  standardized  questionnaire  used  to 
evaluate  the  presence  and  “reasonableness”  of  the 
processes,  activities,  and  resources  needed  to  support 
a  fielded  software  system;  rate  the  software  support 
activity  (SSA)  as  acceptable  or  unacceptable;  and 
provide  feedback  to  the  SSA  on  problem  areas. 

1.2.  Overview  of  the  Guide.  This  guide  is  divided 
as  follows: 

Chapter  1  -  provides  general  information  about  the 
evaluation  and  covers  the  responsibilities  of  the 
AFOTEC  personnel  involved. 

Chapter  2  -  describes  the  philosophy  and  process  of 
evaluating  software  support  resources  to  include 
evaluation  planning,  conduct,  and  reporting. 

Attachment  1  -  Questionnaire.  Questions  were  de> 
rived  from  multiple  sources  (e.g.,  the  Software  Engi¬ 
neering  Institute’s  Capability  Maturity  Model  for 
Software;  various  DoD  and  MIL  standards  and  hand¬ 
books). 

Attachment  2  -  Summary  answer  sheet.  Use  this 
answer  sheet  to  transcribe  your  ratings  and  important 
comments  from  the  questionnaire.  Send  them  to 
AFOTEC/SAS  for  incorporation  into  SAS's  historical 
database. 

1.3.  Overview  of  the  Evaluation. 

1.3.1.  What  are  ^^software  support  resources’’  and 
why  are  they  important?  Software  support  re¬ 
sources  are  the  plans,  processes,  people,  and  physical 
resources  required  to  support  a  software  system  after 
deployment.  Post-deployment  modifications  to  soft¬ 
ware  are  often  necessary  to  correct  errors,  enhance 
system  capabilities,  and  modify  software  to  be  com¬ 
patible  with  changes  in  the  computing  environment. 
Software  support  resources  are  important  because  of 
their  impact  on  software  supportability.  Software 
often  dominates  a  system’s  life-cycle  cost  and  respon¬ 
siveness  to  changing  mission  requirements.  Con¬ 
sequently,  the  Air  Force  invests  billions  of  dollars 
each  year  supporting  deployed  software — ^much  more 


than  is  spent  developing  new  software.  Software 
support  resources  are  one  of  three  major  elements  of 
a  software  system  that  affect  the  ability  of  software 
maintenance  personnel  to  make  changes  (figure  1). 


Life  Cycle  Maintain-  Support 

Processes  abiiity  Resources 


Figure  1-1.  Elements  of  Software  Supportability 

1.3.2.  Why  are  software  support  resources  evalu¬ 
ated  during  OT&E?  Software  supportability  is 
evaluated  as  part  of  the  AFOTEC  assessment  of 
system  suitability.  A  software  system  that  cannot 
continue  to  evolve  to  meet  user  requirements  after  the 
system  is  fielded  is  not  suitable.  Other  software 
evaluations  that  address  suitability  include  the  soft¬ 
ware  support  life-cycle  p  ocess  (volume  2)  and  soft¬ 
ware  maintainability  (volume  3).  The  goal  of  the 
software  support  resources  evaluation  is  to  provide 
decision-makers  with  information  necessary  for 
making  acquisition  decisions  and  to  report  software 
support  deficiencies  to  facilitate  improvements. 

1.3.3.  How  are  software  support  resources  evalu¬ 
ated?  This  evaluation  does  not  directly  measure  the 
capability  of  software  support  resources,  rather  it 
evaluates  the  presence  and  “reasonableness”  of  the 
processes  and  resources  needed  to  support  a  fielded 
software  system.  The  evaluation  focuses  on  software 
support  activity  (SSA)  plans,  processes,  activities, 
and  resources.  Typically  the  software  test  manager 
(STM),  deputy  for  software  evaluation  (DSE)  and/or 
SSR  evaluators  answer  a  standardized  questionnaire 
by  reviewing  documentation  and  interviewing  key 
program  personnel.  Each  question  and  topic  area  is 
rated  as  either  ACCEPTABLE  or  UNACCEPTABLE.  If 
UNACCEPTABLE,  an  impact  rating  is  also  assigned. 
The  overall  score  for  the  system  is  compared  against 
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an  AFOTEC  threshold  to  determine  whether  the 
software  support  resources  are  adequate  or  not. 

1.3.4.  When  are  software  support  resources 
evaluated?  Ideally,  the  SSR  evaluation  should  be 
conducted  continually  throughout  the  engineering  and 
manufacturing  development  (EMD)  phase.  As  the 
software  support  concept  matures,  the  focus  of  the 
evaluation  will  move  from  planning  processes,  to 
actual  plans,  to  an  operating  SSA.  As  an  alternative, 
an  SSR  evaluation  can  be  conducted  in  a  short  period 
of  time  with  a  goal  of  producing  a  report.  Any  unre¬ 
solved  issues  at  the  end  of  dedicated  OT&E  should  be 
documented  in  the  final  OT&E  report. 

1.4.  Personnel  Responsibilities.  The  responsibili¬ 
ties  of  the  STM,  the  test  team  DSE  and  the  SSR 
evaluators  are  described  in  AFOTECPAM  99-102, 
volume  1.  The  following  paragraphs  add  detail  for 
this  specific  evaluation  methodology. 

1.4.1.  STM  Responsibilities.  The  STM  is  respon¬ 
sible  for  developing  the  softwar^^  inputs  to  the  test 
concept  for  the  test  manager.  If  a  DSE  has  already 
been  assigned  to  the  test  program,  then  he/she  will 
assist  in  developing  these  inputs.  The  STM  decides  if 
an  SSR  evaluation  is  necessary  and  when  to 
accomplish  the  evaluation  and  documents  this 
decision  in  his/her  test  concept  inputs.  When  a  test 
team  DSE  is  in  place,  the  STM  provides  the  support 
necessary  for  the  DSE  to  perform  the  evaluation.  The 
STM  also  ensures  the  evaluation  is  conducted  in  the 
manner  described  in  this  pamphlet,  and  ensures  the 
aggregation  strategy  for  the  evaluation  is  followed. 

1.4.2.  DSE  Responsibilities.  The  test  team  DSE  is 
required  to  conduct  the  evaluation  in  the  manner 
described  in  this  pamphlet  and  consistent  with  the 
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planning  performed  by  the  STM.  The  DSE  is  also 
required  to  report  the  results  using  the  aggregation 
strategy  contained  herein.  If  there  is  no  DSE  on  the 
test  team,  the  STM  performs  the  duties  of  both  the 
STM  and  the  DSE. 

1.4.3.  SSR  Evaluator  Responsibilities.  The  SSR 
evaluators,  under  the  supervision  of  the  DSE,  carry 
out  the  evaluation  activities  identified  in  the  opera¬ 
tional  assessment  or  OT&E  test  plan.  These  activi¬ 
ties,  as  they  relate  to  the  SSR  evaluation,  include 
assisting  the  DSE  in  the  collection  and  review  of 
evaluation  materials  (including  software  support  re¬ 
sources  and  computer  system  documentation)  and 
preparation  of  the  OA  or  OT&E  report.  SSR  evalu¬ 
ators  may  be  AFOTEC  resources  (test  team  members) 
or  support  facility  personnel.  Because  the  contribu¬ 
tions  of  the  evaluators  are  most  important  to  the 
quality  of  the  SSR  evaluation  results,  the  following 
qualifications  are  desirable  for  their  participation  in 
the  evaluation: 

1. 4.3.1.  Experience  in  some  phase  of  the  develop¬ 
ment  of  a  software  support  facility  (design,  prototype 
or  operation). 

1. 4.3.2.  Experience  in  the  management  or  use  of  soft¬ 
ware  support  resources. 

1. 4.3.3.  Technical  experience  in  the  hardware  and 
software  areas  of  the  computer  systems  for  which  the 
resources  are  being  evaluated. 

1. 4.3.4.  Familiarity  with  operational  practices  that 
contribute  to  effective  and  efficient  support  of  com¬ 
puter  system  software. 


Chapter  2 

EVALUATION  PHILOSOPHY  AND  PROCESS 


2.1.  Evaluation  Philosophy. 

2.1.1.  The  purpose  of  this  evaluation  is  to  determine 
the  presence  or  absence  of  certain  processes,  activi¬ 
ties,  and  facilities  that  provide  the  SSA's  capability  to 
support  the  system.  Multiple  evaluations  may  be 
necessary  to  help  determine  if  the  SSA  is  progressing 
towards  meeting  its  mission  support  requirements. 
This  will  also  help  identify  problem  areas  and  docu¬ 
ment  the  SSA's  progress.  You  should  also  use  this 
guide  continuously  throughout  the  EMD  phase  to 


review  documentation  as  it  becomes  availaible  and  to 
generate  topics  for  discussion  at  software  meetings, 
technical  interchange  reviews,  design  reviews,  etc. 
Raising  software  support  issues  early  is  the  best  way 
to  improve  processes  and  products  and  avoid  higher 
costs  later. 

2.1.2.  The  support  processes,  activities,  and  facilities 
are  broken  into  six  topics  for  ease  of  evaluation: 


Early  SSA  planning  and  involvement 
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•  SSA  software  project  planning 

•  SSA  software  project  tracking  and  oversight 

•  SSA  software  contract  management 

•  SSA  software  quality  assurance 

•  SSA  software  configuration  management 

2.1.3.  A  Software  Support  Activity  (SSA)  is  defined 
as  the  organizations  responsible  for  managing  and 
performing  the  software  support  effort.  The  same 
organization  may  both  manage  and  perform  the  sup- 
port,  or  software  projects  may  be  contracted  to  a 
maintenance  organization  or  to  the  original  devel¬ 
oper.  When  the  management  and  performance  of 
software  support  are  completed  by  different  organi¬ 
zations,  then  both  must  be  evaluated.  Note  that  the 
assignment  of  a  software  system  to  an  SSA  may  not 
be  a  permanent  assignment,  and  there  may  be  several 
contractors  involved  in  implementing  the  software 
support  concept  for  large  systems. 

2.2.  Evaluation  Process.  The  evaluation  process 
includes  three  distinct  phases: 

2.2.1.  Planning.  You  can  accomplish  the  SSR 
evaluation  any  time  during  a  system's  life  cycle,  but 
the  evaluation  is  more  beneficial  if  conducted  during 
the  EMD  phase.  If  possible,  an  ongoing  SSR 
evaluation  should  be  conducted  throughout  the  EMD 
phase.  To  conduct  an  ongoing  evaluation,  update 
your  response  to  each  question  as  new  information 
becomes  available.  An  ongoing  evaluation  can  help 
determine  problem  areas  early  enough  to  provide 
useful  feedback  for  improving  the  system,  and  can 
help  transition  the  program  when  the  action  officer 
changes.  A  report  can  be  written  any  time  formal 
results  are  required.  An  SSR  evaluation  can  also  be 
conducted  over  a  short  period  of  time  with  the  goal  of 
producing  a  required  report.  Planning  activities 
include  locating  and  acquiring  the  needed  documenta¬ 
tion  and  setting  a  suspense  for  the  reports.  It  is 
important  to  evaluate  all  documentation  used  by  the 
SSA.  The  following  list  identifies  some  applicable 
documents: 

•  Software  Quality  Assurance  (SQA)  Plan 

•  Software  Configuration  Management 
(SCM)  Plan 

•  Software  Maintenance  Plan 

•  Software  Test  Plan/Procedures 

•  System  Safety  Program  Plan  (SSPP) 

•  SSA  Training  Plan 

•  SSA  Coding  Standards 

•  MOAs/MOUs  between  using  commands 
and  SSA 

•  Change  Reporting  Instructions 
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•  Contractor  Work  Breakdown  Structure 

(WBS),  CDRL,  Schedule 

•  Computer  Resources  Life  Cycle  Manage¬ 
ment  Plan  (CRLCMP) 

•  Computer  Resources  Working  Group 

(CRWG)  Charter/Minutes 

•  Computer  Resources  Integrated  Support 

Document  (CRISD) 

•  Integrated  Logistics  Support  Plan  (ILSP) 

•  Test  and  Evaluation  Master  Plan  (TEMP) 

While  this  list  is  not  all-inclusive,  it  does  iden¬ 
tify  the  types  of  documents  that  will  be  necessary.  In 
general,  review  any  document  that  guides  how  the 
software  maintenance  work  will  be  done.  You  should 
plan  on  1  to  2  weeks  for  reviewing  the  documents  and 
understanding  how  the  SSA  works. 

2.2.2.  Conducting  the  Evaluation.  The  evaluation 
is  conducted  through  documentation  reviews  and 
interviews.  Try  to  do  the  document  review  over  the 
course  of  1  to  2  weeks.  This  assumes  you  are  able  to 
get  all  of  the  documents  in  one  place  at  one  time.  If 
you  are  conducting  an  ongoing  evaluation,  review 
documents  as  they  become  available.  It  is  important 
that  you  keep  the  evaluation  results  up-to-date.  When 
reading  the  documents,  you  must  provide  comments/ 
rationde  for  all  applicable  questions.  Jot  down 
anything  that  will  help  explain  your  rating. 

You  should  interview  SSA  personnel,  if 
possible,  to  confirm  some  answers.  Set  up  these 
interviews  to  minimize  impact  on  the  SSA.  Set  aside 
1  or  2  days  to  interview  the  SSA  manager,  lead 
software  engineer,  the  SQA  manager,  the  SCM 
manager,  the  SSA’s  test  manager,  computer  systems 
manager,  a  few  programmers,  and  any  others  deemed 
necessary.  If  you  are  conducting  an  ongoing 
evaluation,  conduct  interviews  at  action  officer-level 
meetings  you  attend. 

After  you  have  completed  your  review  of  all 
appropriate  documents,  interview  the  people  who 
•  actually  (or  potentially)  will  perform  the  work.  Try  to 
determine  if  work  is  done  as  described  in  the 
documented  plans.  Table  2.1  contains  sample 
questions  to  be  considered  before  the  evaluation  is 
complete.  Finalize  your  analysis  after  completion  of 
the  documentation  review  and  interviews. 

You  must  understand  how  the  SSA  addresses 
each  topic  and  then  determine  if  the  process/activity 
is  present  and  reasonable.  If  the  process/activity  is 
present  and  reasonable,  mark  the  question  as 
ACCEPTABLE.  Note  that  ACCEPTABLE  does  not 
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Table  2.1.  Document  Review  Guidelines 


Consider  the  following  when  reviewing  SSR- 
related  documentation: 

•  Is  the  required  information  there? 

•  Is  the  information  understandable? 

•  Is  the  information  consistent  with  every¬ 
thing  else  you've  read? 

•  Is  the  information  traceable  to  predecessor 
documents? 

•  Does  the  information  lend  itself  to  making 
the  support  easier? 

•  Is  the  information  correct? 

•  If  you  deem  a  question  unacceptable,  what 
is  the  impact  of  this  deficiency  on  the 

_ overall  software  support? _ 

necessarily  mean  perfect.  If  the  process/activity  isn't 
present,  or  is  present  but  not  reasonable,  mark  the 
question  as  UNACCEPTABLE  and  then  rate  its  impact. 
The  impact  rating  should  be  the  impact  of  this  defi¬ 
ciency  on  the  overall  software  support  as  described  in 
table  2.2. 


tained  to  the  threshold  of  30.  If  the  sum  is  higher 
than  the  threshold  you  should  give  the  SSR  evaluation 
an  UNACCEPTABLE  rating.  Provide  rationale  to  sup¬ 
port  this  rating. 

2.2.4.  Reporting.  AFOTEC  Instruction  99-101, 
Management  of  Operational  Test  and  Evaluation, 
requires  an  activity  report  on  all  software  evaluations. 
The  requirement  for  format,  submission  frequency, 
content,  and  distribution  of  activity  reports  will  vary 
by  program  and  should  be  defined  in  the  OT&E  plan. 
The  Software  Support  Resources  Evaluation  report 
documents  the  results  of  the  evaluation  for  inclusion 
into  the  OT&E  Final  Report,  and  provides  the  pro¬ 
gram  office  and  the  SSA  feedback  on  strengths  and 
weaknesses.  You  must  describe  deficiencies  to  pro¬ 
vide  the  SSA  with  useful  feedback  on  problem  areas. 
Use  a  statement  similar  to  the  following  in  your 
executive  summary:  “Based  on  the  number  and  im¬ 
pact  of  the  deficiencies  found,  the  program’s  software 
support  resources  are  rated  as  acceptable/unaccept¬ 
able.’*  Plan  on  3  weeks  for  developing  and 
coordinating  your  evaluation  report.  DSE-developed 
evaluation  reports  are  normally  signed  by  the  test 
director  or  detachment  commander. 


Table  2.2.  Operational  Impact  Ratings 


RATING 

DESCRIPTION 

Very  Low 

Some  minor  impact  on  the  SSA’s 
ability  to  support  the  software. 

Low 

There  is  a  reasonable  expectation 
the  deficiency  will  impact  the  qual¬ 
ity  and/or  timeliness  of  post¬ 
deployment  software  releases. 

Moderate 

There  is  a  high  expectation  the  defi¬ 
ciency  will  impact  the  quality  and/or 
timeliness  of  post-deployment  soft¬ 
ware  releases. 

High 

There  is  a  reasonable  expectation 
the  deficiency  will  prevent  the  SSA 
from  meeting  documented  oper¬ 
ational  requirements. 

Very  High 

There  is  a  high  expectation  the 
deficiency  will  prevent  the  SSA 
from  meeting  documented  oper¬ 
ational  requirements. 

2.2.3.  Analyzing  the  Results.  Assign  points  to  each 
unacceptable  question  based  on  its  impact:  VERY 
Low  =  1 ;  Low  =  2;  MODERATE  =  5;  HIGH  =  15;  Very 
High  =  30.  Next,  total  the  number  of  impact  rating 
points.  We  realize  the  impact  ratings  are  subjective, 
so  use  your  best  judgment.  Compare  the  sum  ob- 


Note  to  Test  Team  DSEs:  Please  send  a  draft  copy 
of  all  reports  to  your  SAS  software  test  manager  for 
review  before  you  release  them.  In  addition,  be  sure 
to  provide  a  copy  of  your  SSR  answer  sheet 
(attachment  2). _ 


2.3.  Notes  to  the  Evaluators.  Keep  in  mind  the 
following  points  when  performing  an  SSR  evaluation: 

2.3.1.  The  SSR  evaluation  is  designed  to  evaluate 
processes  and  activities  of  the  SSA — whether  the 
support  activity  is  an  Air  Logistics  Center  (ALC),  the 
user,  or  a  contractor.  It  is  important  to  determine  who 
will  support  the  software  system. 

2.3.2.  You  can  perform  this  evaluation  even  if  the 
SSA  is  not  yet  activated.  If  the  SSA  does  not  come 
bn  line  until  later  in  the  life  cycle,  evaluate  the  proc¬ 
esses  currently  in  use  and  the  future  plans  for  the 
SSA. 

2.3.3.  Remember  you  are  determining  if  the  activities 
or  processes  are  present  and  reasonable  for  the  pro¬ 
gram.  When  completing  the  questionnaire,  avoid 
trying  to  find  out  how  the  SSA  could  be  better.  Your 
job  is  not  to  inspect  their  work.  You  are  there  to 
determine  if  the  SSA  can  support  the  software  system. 
If  you  do  see  potential  improvements,  provide  them 
as  suggestions — not  as  part  of  the  rating. 
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2.3.4.  Bear  in  mind  that  you  have  been  chosen  for  a 
specific  evaluation  based  upon  your  demonstrated 
expertise.  That  expertise  and  professionalism  will  go 


a  long  way  in  providing  the  Air  Force  with  quality 
software  support  resources. 


GEORGE  B.  HARRISON 
Major  General,  USAF 
Commander 
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QUESTIONNAIRE 


Al.l.  Early  Software  Support  Activity  (SSA)  Planning  and  Involvement  The  SSA  must  participate  early  in  the 
acquisition  process.  Software  acquisition  issues  are  important  components  of  the  program  manager's  responsibilities, 
and  acquisition  managers  at  all  levels  should  understand  that  post-deployment  software  support  (PDSS)  cost, 
particularly  software  life  cycle  cost,  is  largely  determined  during  acquisition.  In  addition  to  providing  cost  saving 
PDSS  concept  alternatives,  the  SSA  supports  the  program  manager  by  performing  or  actively  participating  in  many 
other  program  activities. 

A1.2.  Software  Project  Planning.  Software  project  planning  involves  developing  estimates  for  the  work  to  be 
performed,  establishing  the  necessary  commitments,  and  defining  the  plan  to  perform  the  work.  The  SSA  establishes 
a  plan  to  address  the  commitments  to  the  customer  according  to  the  resources,  constraints,  and  capabilities  of  the 
project.  The  plan  provides  the  basis  for  initiating  the  software  maintenance  effort,  testing,  and  managing  the 
progress  of  the  work. 

A1.3.  Software  Project  Tracking  and  Oversight  Software  project  tracking  and  oversight  involve  tracking  and 
reviewing  the  software  accomplishments  and  results  against  documented  estimates,  commitments,  and  plans.  The 
SSA  must  then  adjust  these  based  on  the  actual  accomplishments  and  results.  The  SSA  uses  a  documented  plan  for 
the  software  maintenance  effort  to  track  the  software  maintenance  activities,  communicating  status,  and  revising 
plans.  Software  managers  monitor  the  software  activities  on  a  regular  basis.  The  SSA  conducts  regular  technical 
reviews  and  management  reviews  to  ensure  management  and  staff  are  aware  of  the  software  project's  status  and 
plans,  and  that  issues  receive  appropriate  attention. 

A1.4.  Software  Contract  Management.  Software  contract  management  involves  establishing  commitments  with 
the  contractor  on  the  work  to  be  performed,  coordinating  activities  with  the  contractor,  and  tracking  and  reviewing 
performance  and  results.  The  SSA  establishes  a  documented  agreement  covering  the  technical  and  nontechnical 
requirements  and  is  the  basis  for  managing  the  contract.  The  SSA  conducts  regular  technical  and  management 
reviews  to  ensure  management  and  staff  of  both  organizations  are  aware  of  the  software  status/plans,  and  that  issues 
receive  appropriate  attention.  NOTE:  Skip  this  section  if  no  work  is  to  be  contracted. 

A1.5.  Software  Quality  Assurance  (SQA).  Software  quality  assurance  involves  reviewing  and  auditing  the 
software  products  and  activities  to  ensure  that  they  comply  with  the  applicable  processes  standards,  and  procedures. 
The  SQA  group  provides  the  staff  and  managers  with  the  results  of  the  reviews  and  audits.  A  software  quality 
assurance  function  should  be  required  on  all  projects.  The  SQA  group  is  independent  of  the  software  groups  and 
project  management.  The  SSA  identifies  a  senior  manager  who  is  committed  to  handling  software  quality  issues. 
Where  compliance  issues  exist,  the  SQA  group  works  with  the  appropriate  managers  to  resolve  the  issues. 

A1.6.  Software  Configuration  Management  (SCM).  Software  configuration  management  involves  controlling 
project  baseline  items  (e.g.,  the  project  description,  products,  and  process  specifications)  and  changes  to  them.  SCM 
also  involves  recording  and  reporting  status  and  change  activity  for  these  items.  The  SCM  group  systematically 
controls  these  baseline  items  using  a  defined  change  control  process.  The  SCM  group  can  identify  the  configuration 
(software  and  documentation)  of  a  system,  or  of  any  of  the  controlled  intermediate  or  support  products,  at  any  point 
in  time. 

For  the  purposes  of  the  SSR  questionnaire,  we  used  the  term  “approach”  to  denote  a  process,  a  plan,  or  existing  on¬ 
site  resources.  If  a  plan  for  any  given  activity  isn't  developed  yet,  evaluate  the  process  used  to  develop  such  a  plan. 
If  the  SSA  has  developed  a  plan  specifically  for  this  project,  then  evaluate  that  plan.  If  the  SSA  is  already  operating 
(e.g.,  the  facilities,  hardware,  software,  and  staff  are  in  place),  then  evaluate  the  resources. _ _ 
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I.  Early  SSA  Planning  and  Involvement 


Topic:  Early  SSA  Planning  and  Involvement  Question  No.:  PD-1 

Question:  Early  planning  for  PDSS  is  adequate. 

Discussion:  Ensure: 

□  Early  designation  and  participation  by  the  SSA. 

□  The  program  office  reviews  PDSS  plans  periodically  to  ensure  currency  and  that  assumptions  are  sull 
valid. 

□  The  CRLCMP  and  ILSP  include  the  PDSS  concept  and  SSA  resource  requirements. 

□  The  SSA  actively  participates  in  CRLCMP  development  and  the  CRLCMP  complements  the  ILSP. 

Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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Topic:  Early  SSA  Planning  and  Involvement  Question  No.:  PD-2 

Question:  Acquisition  requirements  imposed  on  the  software  developer  which  facilitate  PDSS  are  identified. 
Discussion:  Ensure: 

□  Specific  hardware  and  software  requirements  are  specified  in  the  development  contract  to  promote  a 
uniform  PDSS  environment  at  the  SSA  (e.g.,  automated  tools,  network  protocols,  document  or  publication 
standards,  configuration  management  forms/documents  or  data  elements. 

□  The  SSA  identified  the  necessary  technical  data  (i.e.,  software  documentation). 

□  The  SSA  and  the  program  office  have: 

•  identified  and  defined  software  quality  requirements. 

•  developed/maintained  standard  techniques  for  software  quality  evaluation. 

•  established  acceptance  criteria. 

□  The  SSA  participated  in  the  evaluation  of  the  program  office's  plans  and  procedures  for  software 
management,  software  engineering,  SCM,  software  corrective  action,  etc. 

□  The  program  office  has  contractually  identified  transition  requirements  (e.g.,  hardware,  networks,  software 
installation  and  test,  security,  maintenance,  training,  transfer  of  software  licensing  agreements,  and 
transition  of  SCM). 

Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Cbmments/Rationale: 
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Topic:  Early  SSA  Planning  and  Involvement  Question  No.:  PD-3 

Question:  The  SSA  actively  participates  in  the  evaluation  of  software  product  supportablility. 

Discussion:  Ensure; 


□  The  SSA  is  actively  involved  in  defining  quality  standards  for  the  software  product. 

□  The  SSA  is  actively  involved  in  defining  the  software  engineering  environments,  and  SSA  resource 
requirements. 

□  The  SSA  examines  software  characteristics,  particularly  correctness,  testability,  and  flexibility. 


NOTE:  Opportunities  for  the  SSA  to  evaluate  software  supportability  include: 

•  Software  product  and  activity  reviews. 

•  Software  technical  reviews  and  audits. 

•  Software  Independent  Verification  and  Validation  activities. 

•  AFOTEC  software  supportability  and  maturity  evaluations. 

•  Formal  qualification  testing. 

•  Government  software  acceptance  evaluations. 


Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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Topic:  Early  SSA  Planning  and  Involvement  Question  No.:  PD-4 

Question:  The  SSA  is  actively  involved  in  assuring  software  quality  and  program  requirements  are  achieved  and 
correctly  implemented. 

Discussion:  Ensure  the  SSA  participates  in: 

□  Authenticating  specifications. 

□  Verifying  requirements. 

□  Evaluating  proposed  software  quality  plans,  records  and  activities. 

Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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Topic:  Early  SSA  Planning  and  Involvement  Question  No.:  PD-5 

Question:  The  SSA  actively  participates  in  transition  planning. 

Discussion:  Ensure  the  SSA  transition  activities  include: 

Q  Participation  on  the  Configuration  Control  Boards. 

□  Acquisition  of  all  needed  resources  used  or  generated  during  software  development. 

□  Installation  and  check  out  of  the  deliverable  software  in  the  support  environment. 

□  Demonstrating  the  deliverable  software  can  be  regenerated  (compiledTlinked/loaded  into  an  executable 
product)  and  maintained  using  available  support  software  and  hardware. 

Ql  Adequately  training  personnel  to  provide  software  support. 

•  □  Turnover,  installation,  checkout,  and  integration  of  any  hardware  or  software  received  ftom  sources  other 

than  the  developing  agency.  .  ,  .  u 

□  Implementation  of  all  required  PDSS  activities  and  capabilities  (e.g.,  problem  replication  and  fault 
isolation,  corrective  action,  software  generation,  integration  and  test,  support  systems,  document 
production). 

□  Approval  and  implementation  of  applicable  software  management  plans  (e.g.,  configuration  management 
plan,  software  quality  plan). 

□  Verification  that  transition  milestones  have  been  correctly  completed  and  all  necessary  resources  are 
available. 

Q  Integration  of  all  PDSS  activities  into  a  cohesive  PDSS  process. 

□  Reporting  software  transfer. 

□  Determination  that  security  requirements  have  been  satisfied. 

□  Determination  that  safety  requirements  have  been  satisfied. 

□  Configuration  management. 

Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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Topic:  Early  SSA  Planning  and  Involvement  Question  No.:  PD-6 

Question:  The  Computer  Resources  Working  Group  (CRWG)  is  chartered  and  operational. 

Discussion:  Ensure  the  CRWG: 

Q  Is  formally  chartered  with  the  coordination  of  the  operating,  supporting,  and  participating  commands  as 
well  as  AFOTEC  and  any  other  OTAs. 

G  Meets  regularly  and  advises  program  management  in  all  areas  relating  to  the  acquisition  and  support  of 
computer  resources. 

□  Develops  and  periodically  updates  the  CRLCMP. 

□  Selects  a  software  support  concept  and  documents  it  in  the  CRLCMP. 

□  Monitors  compliance  of  the  program  with  computer  resources  policy,  plans,  procedures,  and  standards. 

□  Integrates  software  test  activities  with  the  overall  test  program. 

Rating:  ACCEPTABLE  UNACCEPTABLE  +  IMPACT:  VERYLOW 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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II.  SSA  Software  Project  Planning 


Topic:  Software  Project  Planning  Question  No.:  PP-1 

Question:  The  SSA  has  as  a  documented  approach  to  allocate  maintainability  requirements  in  a  consistent  format 
and  to  ensure  the  requirements  are  clearly  stated,  verifiable,  and  testable. 

Discussion:  Ensure  the  approach  requires  the  SSA  to  include  in  program  documentation: 

(NOTE:  If  late  in  the  project's  development  life  cycle  (i.e.,  approaching  OT&E)  evaluate  and  report  if  the  items 
below  are  in  place,  adequate,  and/or  operational.) 

□  The  agreements,  conditions  and  contractual  terms  that  affect  and  determine  the  software  maintenance 
effort  --  deliverables,  delivery  dates,  milestones,  programming  languages  and  software  engineering 
environments. 

□  The  technical  requirements  for  the  software. 

□  The  interface  requirements,  including  the  external  software  interfaces  to  hardware,  other  software  systems 
and  human  interfaces. 

Q  The  criteria  used  to  evaluate  the  software  products  for  acceptance. 

Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Conunents/Rationale: 
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Topic:  Software  Project  Planning  Question  No.:  PP-2 

Question:  The  SSA  develops  the  project's  software  maintenance  plan  according  to  a  documented  approach. 
Discussion:  Ensure  the  approach  requires  the  SSA  to: 

□  Base  the  plan  on  the  approved  work  statements  and  the  approved  allocated  requirements,  and  conforms  to 
customer  and  project  standards. 

□  Negotiate  and  budget  plans  for  involvement  of  the  software  maintenance  group  in  the  activities  of  other 
project  groups  (i.e.,  SQA)  are  negotiated  and  budgeted. 

□  Document  the  agreements. 

The  software  maintenance  plan  should  address  the: 

□  Project  purpose,  scope,  goals  and  objectives. 

□  Identification  and  description  of  the  project’s  software  maintenance  processes. 

□  Identification  of  the  approaches,  methods,  and  standards  for  software  maintenance  and  software 
management. 

□  Software  maintenance  group  participation  in  overall  project  planning. 

□  Identification  of  software  products  to  be  developed  -  products  for  internal  use,  products  for  use  by  other 
product  groups  (i.e.,  SQA),  products  for  delivery  to  the  external  customer. 

□  Size  estimates  of  the  software  products. 

□  Staff  resource  estimates  (skills  and  numbers). 

□  Staff  training  requirements  (both  initial  and  continuation). 

□  Software  project  schedules  and  milestones. 

□  Identification  and  assessment  of  the  project’s  software  risks. 

□  Emergency  action  fixes  and  emergency  releases. 

□  Hardware  and  software  requirements. 

Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Radonale: 
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Topic:  Software  Project  Planning  Question  No.:  PP-3 

Question:  The  SSA  derives  estimates  for  software  size,  maintenance  resources,  costs,  and  risks  according  to  a 
documented  approach. 

Discussion:  Ensure  the  approach  requires  the  SSA  to: 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 


Estimate  software  size  for  all  software  products  and  activities. 

Using  historical  data  where  available. 

Document  size  estimating  assumptions. 

Document,  review,  and  agree  to  size  estimates. 

Relate  estimates  for  software  maintenance  resources  and  costs  to  the  size  estimates  of  the  software 
products. 

Use  objective  productivity  data  (historical  and  current)  from  the  organization's  projects  for  estimates. 

Base  effort/staffing  and  cost  estimates  on  past  experience.  „ 

Document,  review,  and  agree  to  the  estimates  and  assumptions  made  in  deriving  the  estimates. 

Analyze  and  prioritize  the  risks  based  on  their  potential  product  impact. 

Identify  contingencies  for  the  risks. 


Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Conmients/Rationale: 
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Topic:  Software  Project  Planning  Question  No.:  PP-4 

Question:  The  SSA  derives  estimates  for  critical  computer  resources  according  to  a  documented  approach. 
Discussion:  Ensure  the  approach  requires  the  SSA  to:  , 

□  Identify  critical  computer  resources  for  the  project. 

Q  Relate  the  estimates  for  the  computer  resources  to  the  estimates  of  software  product  size,  operational 
processing  load,  and  communications  traffic. 

□  Document,  review,  and  agree  to  estimates  of  critical  computer  resources. 

Rating:  Acceptable  Unacceptable + Impact:  Very  Low 

Low. 

Moderate 

High 

Very  High 


CommentsARationale: 
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Topic:  Software  Project  Planning  Question  No.:  PP-5 

Question:  The  SSA  derives  the  project's  software  schedule  according  to  a  documented  approach. 

Discussion:  Ensure  the  approach  requires  the  SSA  to: 

□  Relate  the  software  schedule  to  the  estimates  for  software  size  and  software  maintenance  resources  and 
costs. 

□  Base  the  software  schedule  on  past  experience. 

□  Relate  the  software  schedule  to  imposed  milestone  dates,  critical  dependency  dates  and  other  constraints. 

□  Ensure  the  software  schedule  activities  and  milestones  are  of  appropriate  duration/time  separation  to 
support  reasonable  accuracy  in  progress  measurement. 

□  Objectively  determine  the  completion  of  software  schedule  activities  and  milestones. 

Q  Document,  review,  and  agree  to  the  software  schedule. 

□  Establish  regularly  scheduled  releases  and  ensure  the  release  schedule  accommodates  the  project’s 
changing  environment. 

Ratine:  Acceptable  Unacceptable + Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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Topic:  Software  Project  Planning  Question  No.:  PP-6 

Question:  The  SSA  prepares  plans  for  the  project’s  software  maintenance  facilities,  environments,  and  support  tools 
according  to  a  documented  approach. 

Discussion:  Ensure  the  approach  requires  the  SSA  to: 

□  Base  estimates  of  capacity  requirements  for  these  facilities,  environments,  and  tools  (i.e.,  software  test 
computers  and  peripherals)  on  the  project's  software  size  and  stability  estimates  and  other  characteristics. 

□  Assign  responsibilities  and  negotiate  commitments  to  procure  or  develop  these  facilities,  environments, 
and  tools. 

□  Review  and  approve  the  plans  for  the  facilities,  environments,  and  tools. 

Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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Topic:  Software  Project  Planning  Question  No.:  PP-7 

Question:  The  SSA  develops  proposed  manning  levels  and  workload  for  this  project  according  to  a  documented 
approach. 

Discussion:  Ensure  the  approach  requires  the  SSA  to: 

Q  Base  manning  levels  on  the  expected  rate  of  change  in  the  software. 

□  Use  software  size  and  cost  estimating  tools  to  help  plan  workload  (Several  tools  are  available  -  REVIC, 
SASET,  COCOMO,  Ada  COCOMO,  SEER-SEM,  SLIM,  CHECKPOINT,  SoftCost-Ada,  SoftCost-R, 
Estimacs,  Price  S,  etc.). 

□  Calibrate  estimating  tools  regularly  using  actual  data  collected  by  the  SSA. 

If  appropriate  for  this  stage  of  the  project's  life  cycle,  answer  the  following: 

□  For  the  staff  currently  maintaining  the  system,  at  least  5  percent  of  the  maintenance  personnel  have  in- 
depth  experience  with  the  system. 

Q  The  average  number  of  years  experience  as  a  maintenance  programmer  for  those  who  maintain  the  system 
is  at  least  2  years. 

□  Software  maintenance  personnel  turnover  per  year  is  less  than  30  percent. 

□  The  proposed  manning  levels  for  this  project  are  sufficient. 

Ratine:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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Topic:  Software  Project  Planning  Question  No.:  PP-8 

Question:  The  SSA  develops  a  training  program  for  this  project  according  to  a  documented  approach. 

Discussion:  Ensure  the  approach  requires  the  SSA  to  address: 

Q  The  minimum  qualifications  for  every  SSA  position. 

□  Continuing  professional  education  to  allow  individuals  to  maintain  currency  and  increase  their  expertise. 

Q  Programmed  funds  for  continued  professional  education. 

Rating:  ACCEPTABLE  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


CommentsfRationale: 
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Topic:  Software  Project  Planning  Question  No.:  PP-9 

Question:  The  SSA  conducts  software  testing  according  to  a  documented  approach. 

Discussion:  Ensure  the  approach  requires: 

□  The  SSA  to  establish  an  independent  test  group. 

Q  Standards  for  lower  level  testing  activities  (e.g.,  code  and  unit  testing)  and  identification  of  test 
responsibilities. 

□  The  test  group  to  develop  the  test  approaches  independently  of  the  maintenance  progralnming  group 
(invalid  assumptions  made  during  code  development  may  be  introduced  if  programmers  develop  test 
approaches). 

□  Documentation  of  test  inputs  and  expected  test  outputs  for  each  test  case  to  enhance  repeatability  and  to 
help  identify  the  cause  of  test  case  failures. 

□  The  test  group  to  conduct  formal  testing  rather  than  the  maintenance  programming  group  (will  help 
eliminate  test  execution  bias). 

□  The  SSA  to  conduct  informal  (lower-level)  testing  to  provide  validity  and  repeatability. 

□  Documentation  of  informal  testing  approaches  and  results  (software  development  folders  provide  evidence 
.  of  lower-level  testing  completion  and  provide  insight  into  the  workings  of  the  software  modules). 

□  The  system  to  undergo  formal  qualification  testing  prior  to  release  to  the  field  (to  prove  changes  don’t 
adversely  impact  the  system). 

Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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III.  SSA  Software  Project  Tracking  and  Oversight 


Topic:  Software  Project  Tracking  and  Oversight  Question  No.:  TO-1 

Question:  The  SSA  tracks  software  maintenance  activities  and  communicates  status  according  to  a  documented 
approach. 

Discussion:  Ensure  the  approach  requires  the  software  maintenance  plan  to  be: 

□  Readily  available  for  use  by  maintainers,  testers,  and  management, 

Q  Regularly  updated  as  the  work  progresses  to  reflect  progress,  incorporate  plan  changes,  and  when  major 
milestones  are  completed. 

□  Reviewed  and  approved  at  each  revision. 

□  Maintained  under  configuration  management. 

Rating:  Acceptable  Unacceptable + Impact:  Very  Low 

Low 

MODERATE 

High 

Very  High 


Comments/Rationale: 
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Topic:  Software  Project  Tracking  and  Oversight  Question  No.:  TO-2 

Question:  The  SSA  has  a  documented  approach  to  track  the  project's  software  size  and  takes  corrective  action  when 
actual  size  differs  significantly  from  estimates. 

Discussion:  Ensure  the  approach  requires  the  SSA  to: 

d  Track  the  sizes  of  all  major  software  products  and  software  activities  (e.g.,  operational  versus  support 
software,  deliverable  versus  nondeliverable  products,  SQA  versus  testing  activities). 

□  Compare  the  actual  size  of  the  code  (generated,  fully  tested,  and  delivered)  to  the  documented  estimates. 

Ql  Refine,  monitor,  and  adjust  on  a  regular  basis  the  overall  projected  size  (estimates  versus  actual). 

Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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Topic:  Software  Project  Tracking  and  Oversight  Question  No.:  TO-3 

Question:  The  SSA  has  a  documented  approach  to  track  the  project's  software  costs  and  takes  corrective  action 
when  actual  costs  differ  significantly  from  estimates. 

Discussion:  Ensure  the  approach  requires  the  SSA  to: 

□  Compare  actual  expenditures  over  time  against  work  completed  to  the  documented  estimates  to  identify 
potential  overruns  and  underruns. 

□  Track  software  costs  and  compares  actual  versus  estimated  costs. 

□  Resolve  changes  to  software  cost  profiles  for  projected  activities  according  to  documented  review 
approaches. 

□  Ensure  status  and  deviations  that  require  management  action  are  reported  to  the  appropriate  managers. 

Ratine:  ACCEPTABLE  UNACCEPTABLE  +  IMPACT:  Very  Low 

Low 

Moderate 

High 

Very  High 


Conunents/Rationale: 
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Topic:  Software  Project  Tracking  and  Oversight  Question  No.:  TO-4 

Question:  The  SSA  has  a  documented  approach  to  track  the  project's  critical  computer  resources  and  takes 
corrective  action  when  resources  used  differs  significantly  from  estimates. 

Discussion:  Ensure  the  approach  requires  the  SSA  to: 

□  Track  the  estimated  capacity  and  use  of  the  project’s  critical  computer  resources  for  each  major  software 
component  as  appropriate  (e.g.,  memory  capacity,  process  use,  channel  capacity). 

□  Resolve  changes  in  estimates  of  critical  computer  resources  that  affect  commitments  with  the  SPO  or  user 
according  to  a  documented  commitment  review  approach. 

Rating:  Acceptable  Unacceptable + Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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Topic:  Software  Project  Tracking  and  Oversight  Question  No.:  TO-5 

Question:  The  SSA  has  a  documented  approach  to  track  the  project's  software  schedule  and  takes  corrective  action 
when  the  project  falls  behind  schedule. 

Discussion:  Ensure  the  approach  requires  the  SSA  to; 

□  Adjust  software  size  and  software  cost  measurements  to  reflect  schedule  adjustments. 

□  Compare  software  units  designed,  coded,  unit  tested,  and  integrated  (including  testing)  into  the  next  higher 
level  component  to  the  documented  plan. 

□  Compare  completion  dates  for  test  case/apprbach  executions  and  the  number  of  executions  completed  to 
the  documented  plan. 

Q  Compare  actual  completion  of  software  activities,  milestones,  and  other  commitments  against  the 
documented  plan. 

Rating;  Acceptable  Unacceptable + Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Conunents/Rationale: 
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Topic:  Software  Project  Tracking  and  Oversight  Question  No.:  TO-6 

Question:  The  SSA  tracks  software  maintenance  activities  according  to  a  documented  approach. 

Discussion:  Ensure  the  approach  requires  the  SSA  to: 

□  Report  and  document  technical  status  activity  on  a  regular  basis, 

□  Compare  system  release  contents  for  successive  builds  to  the  documented  release  plan. 

□  Report  and  document  problems  identified  in  any  of  the  software  products. 

□  Track  problems  and  problem  fixes. 

□  Ensure  the  government  owns  the  appropriate  level  of  data  rights. 

Rating:  Acceptable  Unacceptable + Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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Topic:  Software  Project  Tracking  and  Oversight  Question  No.:  TO-7 

Question:  The  SSA  conducts  formal  reviews  and  software  inspections,  according  to  a  documented  approach,  to 
address  the  accomplishments  and  results  of  the  software  maintenance  effort. 

Discussion:  These  reviews  are  conducted  at  selected  project  milestones,  and  at  the  beginning  and  completion  of 
selected  stages.  Ensure  the  approach  requires  these  reviews  to: 

□  Occur  at  meaningful  points  in  a  project's  schedule. 

□  Be  conducted  with  the  customer  when  appropriate. 

□  Address  the  plans  and  status  of  the  software  maintenance  activities. 

□  Address  the  process  implementations  used  in  the  software  maintenance  and  software  management 
activities. 

□  Result  in  the  identification  and  documentation  of  significant  issues,  action  items,  and  decisions. 

□  Address  the  software  risks. 

□  Result  in  the  refinement  of  the  software  maintenance  plan,  as  appropriate. 

Ensure  the  approach  requires  the  SSA  to: 

□  Conduct  reviews  between  first-line  managers  and  software  maintenance  staff. 

□  Conduct  reviews  between  first-line  managers  and  the  project  software  manager. 

Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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Topic:  Software  Project  Tracking  and  Oversight  Question  No.:  TO- 8 

Question:  The  SSA  develops  a  software  metric  program  according  to  a  documented  approach. 

Discussion:  Software  metrics  provide  a  means  to  instrument  the  software  support  process  and  determine  if  cost, 
schedule,  and  quality  requirements  are  being  met  and  to  facilitate  process  improvement. 

Ensure  the  approach  requires  the  SSA  to: 

Q  Develop  a  well-planned,  documented  program  for  selecting  and  implementing  the  metrics. 

□  Gather  each  metric  for  a  distinct  purpose  and  use. 

□  Document  an  analysis  methodology  to  determine  if  the  metric  is  showing  positive  or  negative  information. 
Q  Invest  an  appropriate  level  of  effort  in  data  collection  for  the  metrics. 

(NOTE:  See  AFP  800-48,  Software  Management  Indicators,  for  candidate  metrics  and  in-depth  details  on  each 
metric) 

Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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Topic:  Software  Project  Tracking  and  Oversight  Question  No.:  TO-9 

Question:  The  SSA  is  acquiring  software  maintenance  facilities,  environments,  and  support  tools  according  to  a 
documented  approach. 

Discussion:  Ensure  the  approach  requires  the  SSA  to; 

□  Provide  adequate  space  for  identified  and  projected  manning,  equipment,  and  tools  (i.e.,  software  test 
computers  and  peripherals). 

□  Identify  and  contract  for  the  software  and  hardware  support  vehicles  (e.g.,  version  updates,  equipment 
maintenance  contracts). 

□  Assign  responsibilities  and  negotiate  commitments  to  procure  or  develop  these  facilities,  environments, 
and  tools. 

□  Identify  and  contract  for  the  needed  software  tools  and  support  licenses. 

□  Document  plans  to  integrate  software  tools  into  the  support  process. 

Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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Topic:  Software  Project  Tracking  and  Oversight  Question  No.:  TO-10 

Question:  The  SSA  develops  a  viable  security  program  for  this  project  according  to  a  documented  approach  (if 
applicable). 

Discussion:  Ensure  the  approach  requires  the  SSA  to: 

□  Document  the  program-specific  security  requirements. 

□  Incorporate  DoD  and  Air  Force  standards  and  security  guidance  for  handling,  securing,  and  destroying 
classified  material. 

□  Approve  security  practices  and  procedures  for  each  computer  system. 

□  Have  the  facilities  and  destruction  equipment  available  for  classified  material  storage  and  destruction. 

Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 

Comments/Rationale: 
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Topic:  Software  Project  Tracking  and  Oversight  Question-No.:  TO- 11 

Question:  The  SSA  develops  a  viable  safety  program  for  this  project  according  to  a  documented  approach  (if 
applicable). 

Discussion:  Ensure  the  approach  requires  the  SSA  to: 

□  Identify  the  safety-critical  software  components  of  the  system. 

□  Incorporate  DoD  and  Air  Force  standards  and  safety  guidance  for  modifying,  testing,  and  qualifying 
safety-critical  software. 

□  Maintain  a  software  Safety  Data  Library  (SDL)  with  hazard  analyses  for  the  safety-critical  components. 

Ratine:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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TV.  SSA  Software  Contract  Management 
(Only  applicable  when  the  SSA  contracts  out  work) 


Topic:  Software  Contract  Management  Question  No.:  SM-1 

Question:  The  SSA  defines  and  plans  the  contracted  work  according  to  a  documented  approach. 

Discussion:  Ensure  the  approach  requires  the  SSA  to: 

□  Select  the  functions  to  be  contracted  to  match  the  special  skills  and  capabilities  of  potential  contractors. 

Q  Derive  the  contract  statement  of  work,  standards,  and  approaches  from  the  software  requirements  and  the 
software  maintenance  plan. 

Q  Prepare,  review,  approve,  and  maintain  the  contract  statement  of  work. 

□  Establish  an  appropriate  metrics  program  to  monitor  the  cost,  schedule,  and  quality  of  the  contractors 
.  work. 

□  Review  and  approve  the  contractor's  software  maintenance  plan. 

□  Establish  documented  communication  channels  with  the  contractor  to  report  requirements  changes, 
software  trouble  reports,  and  new  release  actions. 

Ratine:  Acx:eptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Conunents/Rationale: 
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Topic:  Software  Contract  Management  Question  No.:  SM-2 

Question:  SSA  management  regularly  conducts  status/coordination  reviews  with  the  contractor's  management 
according  to  a  documented  approach. 

Discussion:  Ensure  the  approach  requires  the  SSA  to: 

Q  Provide  the  contractor  with  appropriate  visibility  of  the  needs  and  desires  of  the  product’s  end  users  and 
customer. 

□  Review  the  contractor's  technical,  cost,  staffing,  and  schedule  performance  against  the  contractor's 
software  maintenance  plan. 

□  Review  the  use  of  critical  computer  resources. 

.  □  Address  critical  dependencies  and  commitments  between  groups  and  between  the  SSA  and  the  contractor. 
Q  Address  any  contract  nonconformance  issues  and  problems. 

G  Address  project  risks  and  review  action  items. 

Ratine:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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Topic:  Software  Contract  Management  Question  No.:  SM-3 

Question:  The  SSA  holds  periodic  technical  reviews,  interchanges,  and  formal  reviews  with  the  contractor 
according  to  a  documented  approach. 

Discussion:  Ensure  the  approach  requires  the  SSA  (at  these  reviews)  to: 

□  Address  the  technical  activities  and  resolve  technical  issues. 

□  Address  the  contractor’s  commitments  for,  plans  for,  and  status  of  software  maintenance  activities  and  the 
corresponding  process  implementations. 

□  Address  software  risks. 

Q  Refine  the  contractor’s  software  maintenance  plan,  as  appropriate. 

Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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Topic:  Software  Contract  Management  Question  No.:  SM-4 

Question:  The  SSA  conducts  acceptance  testing  as  part  of  delivery  of  the  contractor's  products  according  to  a 
documented  approach. 

Discussion:  Ensure  the  approach  requires  the  SSA  to: 

□  Define,  review,  and  approve  the  acceptance  procedures  and  criteria  for  each  product. 

□  Documents  the  results  of  the  acceptance  tests. 

□  Establishes  an  action  plan  for  any  product  that  does  not  pass  acceptance  testing. 

Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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V,  SSA  Software  Quality  Assurance 


Topic:  Software  Quality  Assurance  Question  No.:  QA-1 

Question:  The  SSA  prepares  an  SQA  plan  for  each  software  project  according  to  a  documented  approach. 

Discussion:  Ensure  the  approach  requires  the  SQA  group  to: 

G  Develop  the  plan  in  conjunction  with  the  project’s  software  managers  and  software  maintenance  task 
leaders. 

□  Get  other  project  groups  (e.g.,  test  group)  to  review  and  agree  to  the  SQA  plan. 

Also,  ensure  the  approach  requires  the  SQA  to  address: 

G  Responsibilities  and  authority  of  the  SQA  group. 

G  Resource  requirements  for  the  SQA  group  (including  staff,  tools,  and  facilities). 

G  SQA  group’s  participation  in  establishing  the  product’s  plan  and  process  baseline. 

G  Product  and  process  evaluations,  audits,  and  reviews  to  be  performed  by  the  SQA  group. 

Q  Project  standards  and  procedures  used  as  the  basis  for  the  SQA  group's  reviews  and  audits. 

□  Documentation  SQA  is  required  to  produce. 

Q  Methodology  and  frequency  of  providing  feedback  to  other  project  groups  on  SQA  audits  and  reviews. 

Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Conunents/Rationale: 
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Topic:  Software  Quality  Assurance  Question  No.:  QA-2 

Question:  Nonproject  management  monitors  the  activities  of  the  SQA  group  according  to  a  documented  approach. 

Discussion:  Ensure  the  approach  requires  management  to: 

□  Review  and  audit  the  SQA  records  and  activities. 

Q  Identify  and  implement  corrective  actions,  as  appropriate. 

Rating:  Acceptable  Unacceptable + Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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Topic:  Software  Quality  Assurance  Question  No.:  QA-3 

Question:  The  SQA  group  reviews  representative  samples  of  deliverable  and  designated  nondeliverable  software 
products  and  other  maintenance  activities  according  to  a  documented  approach  to  ensure  compliance  with  the 
designated  process  requirements. 

Discussion:  Ensure  the  approach  requires  the  SQA  group  to: 

Q  Evaluate  the  deliverable  products  prior  to  delivery  to  the  customer. 

□  Evaluate  the  products  against  the  appropriate  software  standards,  practices,  and  requiremeiits. 

□  Identify/report  deviations  and  product  deficiencies. 

□  Validate  the  contents  of  each  formal  release. 

Rating:  Acceptable  Unaccept  able  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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Topic:  Software  Quality  Assurance  Question  No.:  QA-4 

Question:  The  SQA  group  documents  and  resolves  deviations  in  the  software  maintenance  activities  according  to  a 
documented  approach. 

Discussion:  Ensure  the  approach  requires  the  SQA  group  to: 

Q  Resolve  deviations  from  the  software  maintenance  plan  and  the  appropriate  standards  and  procedures  with 
appropriate  software  managers. 

G  Report  any  unresolved  or  noncompliance  items  to  the  designated  senior  manager. 

G  Review  noncompliance  items  periodically  until  resolved. 

G  Maintain  the  documentation  of  noncompliance  items  under  configuration  management. 

Rating:  ACCEPTABLE  UNACCEPTABLE  +  IMPACT:  VERY  LOW 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 


AFOTECPAM  99-102,  Volume  5  Attachment  1  28  August  1995 


Topic:  Software  Quality  Assurance  Question  No.:  QA-5 

Question:  The  SQA  group  conducts  regular  reviews  of  its  activities  and  findings  according  to  a  documented 
approach. 

Discussion:  Ensure  the  approach  requires  the  SQA  group  to: 

□  Conduct  regular  reviews  of  its  activities  and  findings  with  customer  personnel. 

□  Conduct  peer  reviews. 

□  Regularly  report  the  results  of  its  reviews  and  audits  to  the  software  maintenance  staff  and  managers. 

Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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VI.  SSA  Software  Configuration  Management 


Topic:  Software  Configuration  Management  Question  No.:  CM-1 

Question:  The  SSA  develops  an  SCM  plan  for  each  software  project  according  to  a  documented  approach. 
Discussion:  Ensure  the  approach  requires  the  SCM  group  to: 

□  Address  the  activities  to  be  performed,  the  schedule  of  activities,  the  assigned  responsibilities,  and  the 
resources  required  (including  staff,  tools  and  computer  facilities). 

G  Address  the  SCM  requirements  and  activities  to  be  performed  by  the  software  maintenance  group  and 
other  groups  (e.g.,  test  group). 

□  Develop  the  plan  before  the  software  maintenance  activities  begin. 

G  Review  and  coordinate  the  plan  (all  affected  groups/individuals). 

G  Maintain  the  plan  under  configuration  control. 

G  Use  the  plan  as  the  basis  for  performing  SCM  activities. 

Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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Topic:  Software  Configuration  Management  Question  No.:  CM-2 

Question:  The  SSA  establishes  a  configuration  management  library  system  as  a  repository  for  the  software 
baselines  according  to  a  documented  approach. 

Discussion:  Ensure  the  approach  requires  the  library  system  to: 

□  Provide  for  the  storage  and  retrieval  of  configuration  items  and  their  configuration  components. 

□  Help  enforce  product  standards  (e.g.,  naming  and  format)  of  configuration  items  and  their  configuration 
components. 

□  Provide  for  the  storage  and  recovery  of  archive  versions  of  configuration  items  and  their  configuration 
components. 

Q  Help  ensure  correct  creation  of  software  baseline  products. 

□  Provide  for  the  storage,  update,  and  retrieval  of  SCM  records. 

□  Produce  SCM  reports. 

Rating:  ACCEPTABLE  UNACCEPTABLE  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Conunents/Rationale: 
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Topic:  Software  Configuration  Management  Question  No.:  CM-3 

Question:  The  SSA  places  configuration  items  (software  engineering  products,  process  specifications,  and  software 
maintenance  products)  under  configuration  management  according  to  a  documented  approach. 

Discussion:  Ensure  the  approach  requires  the  SCM  group  to: 

G  Specify  the  characteristics  of  each  configuration  item, 

□  Specify  the  software  baseline  to  which  each  configuration  item  belongs. 

□  Define  the  person  responsible  for  each  configuration  item. 


Examples  of  project  configuration  items  include: 

•  Software  requirements  specifications,  software  designs,  software  code  units. 

•  Software  test  approaches,  software  system  build  for  the  software  test  activity. 

•  Software  system  build  for  delivery  to  the  customer. 

•  Process  specifications,  specifications  for  standards,  and  approaches. 

•  Compilers,  test  data,  and  other  support  tools. 


Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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Topic:  Software  Configuration  Management  Question  No.:  CM-4 

Question:  The  SCM  group  follows  a  documented  approach  to  control  changes  to  configuration  items. 

Discussion:  Ensure  the  approach  requires  the  SCM  group  to: 

□  Follow  established  controls  to  ensure  that  configuration  items  are  checked  in/out  in  a  manner  that 
maintains  the  correctness  and  integrity  of  the  software  baseline  library. 

Q  Perform  reviews  and  identify  regression  test  requirements  to  ensure  changes  have  not  caused  unintended 
effects  on  the  product. 

Q  Audit  revised  configuration  items  to  ensure  they  are  prepared  according  to  SCM  standards  (e.g.,  naming 
standards,  format  standards,  version  number  and  change  history  standards). 

Q  Only  enter  configuration  items  accepted  by  the  Software  Configuration  Control  Board  (SCCB)  into  the 
software  baseline  library. 

G  Present  detailed  cost  and  schedule  information  before  SCCB  approval. 

Rating:  ACCEPTABLE  UNACCEPTABLE  +  IMPACT:  VERY  LOW 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 
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Topic:  Software  Configuration  Management  Question  No.:  CM-5 

Question:  The  SCM  group  follows  a  documented  approach  to  create  and  control  the  release  of  software  baseline 
products,  and  to  record  the  status  of  configuration  items  and  change  requests. 

Discussion:  Ensure  the  approach  requires  the  SCM  group  to: 

□  Record  configuration  management  actions  in  sufficient  detail  so  the  software  baseline  contents  and  status 
are  known  and  previous  versions  can  be  recovered. 

□  Maintain  the  current  status  and  history  of  the  software  baselines. 

G  Uniquely  identify  each  software  baseline. 

Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Conunents/Rationale: 
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Topic:  Software  Configuration  Management  Question  No.:  CM-6 

Question:  The  SCM  group  follows  a  documented  approach  that  requires  standard  reports  to  document  the  SCM 
activities  and  the  contents  of  the  software  baseline. 

Discussion:  Ensure  the  approach  requires  the  SCM  group  to  include  in  these  reports  the: 

□  SCCB  meeting  minutes. 

□  Change  request  summary  and  status. 

□  Trouble  report  summary  and  status  (including  fixes). 

□  Summary  of  changes  made  to  the  software  baselines. 

□  Revision  history  of  configuration  items. 

Q  Software  baseline  status. 

Q  Findings  of  software  baseline  audits. 

□  Distribution  list  (affected  groups  and  individuals). 

Rating:  Acceptable  Unacceptable + Impact;  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationaie: 
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Topic:  Software  Configuration  Management  Question  No.:  CM-7 

Question:  The  SCM  group  follows  a  documented  approach  to  prepare  for,  conduct,  report  results  from,  and  track 
action  items  from  software  baseline  audits. 

Discussion:  Ensure  the  approach  requires  the  auditors  to: 

□  Assess  the  integrity  of  software  baselines. 

Q  Review  the  structure  and  facilities  of  the  library  system  for  configuration  management. 

□  Verify  the  completeness  and  correctness  of  the  library  contents. 

Q  Determine  if  the  SCM  process  is  followed. 

Rating:  Acceptable  Unacceptable  +  Impact:  Very  Low 

Low 

Moderate 

High 

Very  High 


Comments/Rationale: 


Program  Name: 


DSE  Name/Phone: 
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SUMMARY  ANSWER  SHEET 


STM  Name: 


TOPIC  AREA 


Early  SSA  Planning  and  Involvement 


Software  Configuration  Management 


Overall  Rating: 


Acceptable 


Unacceptable 


Impact  Rating  Points 


When  completed,  fax  this  sheet  to  AFOTEC/SAS,  DSN  246-5145,  Commercial  (505)  846-5145.  Please  attach  a 
summary  of  your  comments  and  rationale. 


